feat: cluster autoscaler #251
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #251 +/- ##
=======================================
Coverage 53.83% 53.83%
=======================================
Files 27 27
Lines 1566 1566
=======================================
Hits 843 843
Misses 673 673
Partials 50 50 ☔ View full report in Codecov by Sentry. |
eljohnson92
reviewed
Apr 15, 2024
| 1. Set up autoscaling environment variables | ||
| ```sh | ||
| export CLUSTER_AUTOSCALER_VERSION=v1.29.0 | ||
| export WORKER_MACHINE_MIN="\"1\"" |
There was a problem hiding this comment.
could we move this string escaping into the template instead of the variable?
There was a problem hiding this comment.
I don't think escaping should be needed, it's already quoted in the template
There was a problem hiding this comment.
I can give this another shot, but envsubst didn't play well with quoting numbers as strings in the templating. It was either stripping them too aggressively or not enough.
There was a problem hiding this comment.
The issue here seems to be how clusterctl generate generates templates in conjunction with envsubst. For what I can see, it:
- Renders the templates and then validates the Kubernetes YAML
- Passes the generated YAML through
envsubst
envsubst substitutes both ${var} and "${var}" so you need to explicitly specify the shell variable as a "string".
I've slightly modified the documentation commands to remove the escapes and also provided defaults for these values.
eljohnson92
reviewed
Apr 15, 2024
eljohnson92
reviewed
Apr 15, 2024
cbzzz
force-pushed
the
feat.autoscaling
branch
2 times, most recently
from
8e75985
to
1c70a57
Compare
AshleyDumaine
approved these changes
Apr 17, 2024
eljohnson92
approved these changes
Apr 17, 2024
Adds a new cluster-autoscaler flavor that provides an autoscaling add-on for workload cluster nodes via [Cluster Autoscaler](https://www.github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#cluster-autoscaler). Due to constraints with the Kubernetes RBAC system (i.e. [roles cannot be subdivided beyond namespace-granularity](https://www.github.com/kubernetes/kubernetes/issues/56582)), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters. Currently, the Cluster Autoscaler reuses the `${CLUSTER_NAME}-kubeconfig` Secret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key named `value`. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as the `cluster-admin` role. See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster
AshleyDumaine
pushed a commit
that referenced
this pull request
Apr 19, 2024
* feat: add cluster autoscaler flavor Adds a new cluster-autoscaler flavor that provides an autoscaling add-on for workload cluster nodes via [Cluster Autoscaler](https://www.github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#cluster-autoscaler). Due to constraints with the Kubernetes RBAC system (i.e. [roles cannot be subdivided beyond namespace-granularity](https://www.github.com/kubernetes/kubernetes/issues/56582)), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters. Currently, the Cluster Autoscaler reuses the `${CLUSTER_NAME}-kubeconfig` Secret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key named `value`. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as the `cluster-admin` role. See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster * docs: add cluster autoscaler flavor
amold1
pushed a commit
that referenced
this pull request
May 17, 2024
* feat: add cluster autoscaler flavor Adds a new cluster-autoscaler flavor that provides an autoscaling add-on for workload cluster nodes via [Cluster Autoscaler](https://www.github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#cluster-autoscaler). Due to constraints with the Kubernetes RBAC system (i.e. [roles cannot be subdivided beyond namespace-granularity](https://www.github.com/kubernetes/kubernetes/issues/56582)), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters. Currently, the Cluster Autoscaler reuses the `${CLUSTER_NAME}-kubeconfig` Secret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key named `value`. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as the `cluster-admin` role. See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster * docs: add cluster autoscaler flavor
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds a new cluster-autoscaler flavor that provides an add-on to autoscale workload cluster nodes via Cluster Autoscaler.
Testing
Ready, download the kubeconfig file and deploy any workload.$ KUBECONFIG=/tmp/${CLUSTER_NAME}-kubeconfig kubectl scale deployment/nginx --replicas 2MachineSets,MachineDeployments, orMachinePoolsto meet scheduling requirements.$ KUBECONFIG=/tmp/${CLUSTER_NAME}-kubeconfig kubectl delete deployment/nginxMachineSets,MachineDeployments, orMachinePools. You may need to restart the Cluster Autoscaler with the--scale-down-unneeded-time=1ssetting for a quicker reaction time.Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
Special notes for your reviewer:
Due to constraints with the Kubernetes RBAC system (i.e. roles cannot be subdivided beyond namespace-granularity), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters.
Currently, the Cluster Autoscaler reuses the
${CLUSTER_NAME}-kubeconfigSecret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key namedvalue. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as thecluster-adminrole (and might be insecure idk 🙈).See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster
TODOs: